Papers with expert selection
ORMind: A Cognitive-Inspired End-to-End Reasoning Framework for Operations Research (2025.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown promising results in various domains, but their practical application in industry-relevant operations research presents significant challenges and opportunities. |
| Approach: | They propose a cognitive-inspired framework that enhances optimization through counterfactual reasoning . they use a workflow that transforms requirements into mathematical models and executable solver code . |
| Outcome: | Experiments show that ORMind outperforms existing methods in the NL4Opt dataset and ComplexOR dataset. |
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment (2026.findings-acl)
Copied to clipboard
Tianyu Dong, Yangyang Liu, Jiang Zhou, Xinwei Wu, Xiaohu Zhao, Hao Wang, Heng Liu, Linlong Xu, Longyue Wang, Weihua Luo, Shaolin Zhu, Deyi Xiong
| Challenge: | Low-resource language tokens are often routed to different experts than those activated by high-resourced inputs, which hinders their efficacy in multilingual contexts. |
| Approach: | They propose a framework to transfer specialized capabilities from high-resource languages as anchors to low-resourced languages by using a symmetric Jensen-Shannon constraint. |
| Outcome: | The proposed framework outperforms standard instruction tuning on 5 low-resource languages and 3 benchmarks. |
Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have seen remarkable progress for medical decision-making, however, they are designated for specific classification or generative tasks and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing. |
| Approach: | They propose a framework that tackles discriminative and generative multimodal medical tasks using multimodal alignment, instruction tuning and routing. |
| Outcome: | The proposed model can achieve superior performance to or on par with state-of-the-art baselines while only requiring 30%-50% of activated model parameters. |
CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work has begun to address routing instability in VQA models by grouping similar concepts or routing based on examples. |
| Approach: | They propose a Concept-Guided Routing framework which incorporates semantics of the answer options to guide expert selection in the training phase. |
| Outcome: | The proposed framework delivers strong performance across multiple VQA tasks, demonstrating the effectiveness of the proposed framework. |
Glider: Global and Local Instruction-Driven Expert Router (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for routing-based expert models favor generalization over performance on held-in tasks. |
| Approach: | They propose a global and local instruction driven expert router that leverages recent LLMs' semantic reasoning capabilities to generate task-specific instructions from the input query. |
| Outcome: | The proposed method improves held-in performance while maintaining strong generalization on held-out tasks. |
From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing parameter-efficient approaches to multimodal Continual Instruction Tuning suffer from knowledge interference and inefficient capacity expansion, limiting scalability. |
| Approach: | They propose a framework for multimodal Continual instruction tuning that decomposes adaptation weights into a globally shared pool of orthonormal bases to capture task-invariant knowledge. |
| Outcome: | Experiments show that MoBLoRA outperforms state-of-the-art methods while maintaining superior parameter efficiency. |
HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for enhancing performance through increased use of expert knowledge often result in diminishing sparsity during expert selection. |
| Approach: | They propose a framework that integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning. |
| Outcome: | The proposed framework outperforms existing methods under identical conditions concerning the number of experts. |
MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts (2025.findings-emnlp)
Copied to clipboard
| Challenge: | MultiPL is a special case of multiple natural languages and requires limited computational resources to generate multilingual code. |
| Approach: | They propose to extend LLMs by combining two paired experts to optimize expert selection at token and segment levels. |
| Outcome: | The proposed extension improves the performance of the base LLMs while retaining the most popular ones using limited computational resources. |
Harder Task Needs More Experts: Dynamic Routing in MoE Models (2024.acl-long)
Copied to clipboard
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, Yansong Feng
| Challenge: | Unlike existing MoE approaches that rely on fixed TopK Routing, our dynamic expert selection framework dynamically allocates experts based on the confidence level in expert selection for each input. |
| Approach: | They propose a dynamic expert selection framework that dynamically allocates experts based on the confidence level in expert selection for each input. |
| Outcome: | The proposed method achieves an average improvement of 0.7% with less than 90% activated parameters and outperforms dense models in QA and machine translation tasks. |
THOR-MoE: Hierarchical Task-Guided and Context-Responsive Routing for Neural Machine Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing sparse Mixture-of-Experts (MoE) solutions may lead to sub-optimal performance . thor-moe uses domain/linguistics-specific knowledge, but lacks context-responsive routing policies . |
| Approach: | They propose a sparse Mixture-of-Experts (MoE) solution which uses task knowledge of NMT into MoE and provides hierarchical task-guided and context-responsive routing policies. |
| Outcome: | thor-MoE can achieve an average improvement of 0.75 BLEU with less than 22% activated parameters on multi-domain translation tasks. |